The present invention relates generally to networking technology. More specifically, the present invention relates to the caching of data objects to accelerate access to, for example, the World Wide Web. Still more specifically, the present invention provides methods and apparatus by which caching systems may be made to coexist with servers which require user authentication for access.
Generally speaking, when a client platform communicates with some remote server, whether via the Internet or an intranet, it crafts a data packet which defines a TCP connection between the two hosts, i.e., the client platform and the destination server. More specifically, the data packet has headers which include the destination IP address, the destination port, the source IP address, the source port, and the protocol type. The destination IP address might be the address of a well known World Wide Web (WWW) search engine such as, for example, Yahoo, in which case, the protocol would be TCP and the destination port would be port 80, a well known port for HTTP and the WWW. The source IP address would, of course, be the IP address for the client platform and the source port would be one of the TCP ports selected by the client. These five pieces of information define the TCP connection.
Given the increase of traffic on the World Wide Web and the growing bandwidth demands of ever more sophisticated multimedia content, there has been constant pressure to find more efficient ways to service data requests than opening direct TCP connections between a requesting client and the primary repository for the desired data. Interestingly, one technique for increasing the efficiency with which data requests are serviced came about as the result of the development of network firewalls in response to security concerns. In the early development of such security measures, proxy servers were employed as firewalls to protect networks and their client machines from corruption by undesirable content and unauthorized access from the outside world. Proxy servers were originally based on Unix machines because that was the prevalent technology at the time. This model was generalized with the advent of SOCKS which was essentially a daemon on a Unix machine. Software on a client platform on the network protected by the firewall was specially configured to communicate with the resident daemon which then made the connection to a destination platform at the client""s request. The daemon then passed information back and forth between the client and destination platforms acting as an intermediary or xe2x80x9cproxyxe2x80x9d.
Not only did this model provide the desired protection for the client""s network, it gave the entire network the IP address of the proxy server, therefore simplifying the problem of addressing of data packets to an increasing number of users. Moreover, because of the storage capability of the proxy server, information retrieved from remote servers could be stored rather than simply passed through to the requesting platform. This storage capability was quickly recognized as a means by which access to the World Wide Web could be accelerated. That is, by storing frequently requested data, subsequent requests for the same data could be serviced without having to retrieve the requested data from its original remote source. Currently, most Internet service providers (ISPs) accelerate access to their web sites using proxy servers.
A similar idea led to the development of network caching systems. Network caches are employed near the router of a network to accelerate access to the Internet for the client machines on the network. An example of such a system is described in commonly assigned, copending U.S. patent application Ser. No. 08/946,867 for METHOD AND APPARATUS FOR FACILITATING NETWORK DATA TRANSMISSIONS filed on Oct. 8, 1997, the entire specification of which is incorporated herein by reference for all purposes. Such a cache typically stores the data objects which are most frequently requested by the network users and which do not change too often. Network caches can provide a significant improvement in the time required to download objects to the individual machines, especially where the user group is relatively homogenous with regard to the type of content being requested. The efficiency of a particular caching system is represented by a metric called the xe2x80x9chit ratioxe2x80x9d which is a ratio of the number of requests for content satisfied by the cache to the total number of requests for content made by the users of the various client machines on the network. The hit ratio of a caching system is high if its xe2x80x9cworking setxe2x80x9d, i.e., the set of objects stored in the cache, closely resembles the content currently being requested by the user group.
The network cache described in the above-referenced patent application operates transparently to the client network. It accomplishes this in part by xe2x80x9cspoofingxe2x80x9d the server from which content is requested. That is, if the requested content is in the cache it is sent to the requesting client platform with a header indicating it came from the server having the original content. Even where the requested content is not in the cache, the cache retrieves the original content from the server for which the request was intended, stores it, and then transmits the content from the cache to the requesting client, again indicating that the transmitted data are from the originating server.
As will be understood, some web servers only allow access to real clients. That is, such servers will not transmit requested content in response to a request from a network cache. Only direct requests from the client are honored. Thus, a connection from a cache is rejected and the request is either sent back with an appropriate message in the HTTP header, or the request is simply not answered. Unfortunately, a subsequent request for the same information will go through the same cache with a similar end result. This problem may be solved for a particular cache by configuring the associated router to bypass requests corresponding to certain client/destination pairs as identified by the packet""s HTTP header. That is, the system administrator can add access control lists (ACLs) into the router such that data requests which have previously been identified may be passed through the router without being routed through the associated cache.
However, while this may prove somewhat effective in limited circumstances, it destroys the transparency with which the cache is intended to operate. That is, the system administrator needs to monitor rejected requests and manually reconfigure the router, while users on the client network experience, at least temporarily, frustrating limitations on access to desired content until the router ACL is appropriately modified. Moreover, such a solution cannot work in multi-layer networks which do not share administration. As will be appreciated, this is a significant limitation in that this describes most of the world""s networking infrastructure.
The problem with the multi-layer or hierarchical network is that there are likely to be more than one cache in between the requesting client and the destination server storing the requested content. Thus, unless each of the upstream caches and/or routers are configured to bypass certain requests, the connection will continue to be rejected until all of the independent reconfigurations occur. This is clearly not an acceptable solution.
It is therefore desirable that a technique is provided by which requests to servers requiring real client access may be made to bypass all upstream network caches in a manner which is transparent to both users and network administrators.
According to the present invention, methods and apparatus are provided which enable caching systems in hierarchical networks to recognize data requests headed for destination servers requiring real client access, and to pass such requests through without engaging in the standard caching protocol. The process by which this is accomplished is transparent to the requesting client platform and the system administrator and therefore preserves one of the key features of most caching systems.
When a client platform initially transmits a request specifying a destination platform which requires real client access, an upstream caching system comprising a cache-enabled router and a network cache handles the request as it would any other request. That is, if the request meets certain criteria, e.g., the packet specifies port 80 as the destination port, the router sends it to the associated cache which then determines whether the requested content is present in the cache. Obviously, because of the nature of the destination platform, the requested content is not likely to be in the cache. The cache then attempts to establish a connection to the destination server to retrieve the content.
In attempting to establish the connection to the destination server, the cache crafts a request in which the original client platform from which the request originated is identified. According to a specific embodiment, this information is added to the HTTP header. As will become apparent, the insertion of this identifying information facilitates operation of the invention in a hierarchical environment. Any upstream caching system will handle the modified request according to its standard protocol.
Ultimately, the attempted connection with the destination server by the last cache in the upstream path is rejected. The destination server responds to the last cache with an appropriate message indicating, for example, that the request requires authentication or that authentication had failed. The cache sends a message to its associated router instructing it not to redirect any further requests from the originating client to the destination server, and an entry is made in a table of client/server pairs for which requests are to be bypassed. The cache then sends a message to the originating client platform instructing it to resend the request to the same destination platform. Any intervening downstream caching systems receive this message, add the client/server pair to a resident bypass table, and transmit the resend message to the originating client platform.
In response to the resend message, the client platform retransmits the original request to the same destination platform. For this retransmission, each of the upstream caching systems now recognizes the request as one which should be passed through the cache by reference to its resident bypass table. In this way, the request is able to make it all the way to the specified destination where it is handled appropriately.
Thus, the present invention provides methods and apparatus for routing a data request received by a caching system. The caching system includes a router and a cache, and the data request identifies a source platform, a destination platform, and requested data. Where the source and destination platforms correspond to an entry in a list automatically generated by the caching system, the data request is transmitted without determining whether the requested data are stored in the cache.
According to a specific embodiment of the invention, when it is determined that the requested data are not in the cache, an attempt to establish a connection between the cache and the destination platform is made. Upon receiving notification that the connection has failed, an entry corresponding to the source and destination platforms is automatically stored in a list generated by the caching system. The source platform is then prompted to transmit a second data request for the requested data. In response to the entry in the list, the second data request is passed through the caching system without determining whether the requested data are stored in the cache.
According to another specific embodiment, the data request has a header associated therewith containing a data field. Where the data field corresponds to a first entry in a first list associated with caching system, a second entry corresponding to the source and destination platforms is automatically stored in a second list generated by the caching system. The source platform is then prompted to transmit a second data request for the requested data. In response to the second entry in the second list, the second data request is passed through the caching system without determining whether the requested data are stored in the cache.
A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings.